In digital imaging, it is desirable to capture an image sequence having high image quality, high spatial resolution and high temporal resolution, also referred to as frame rate. With many current image sequence capture devices, however, it is not possible to obtain such high quality image sequences. In many cases, one of the desired image sequence attributes is obtained at the expense of the others. For example, in a conventional image sequence capture device, the exposure duration for a given image is limited by the frame rate. The higher the frame rate, the shorter each image exposure must be. In a low-light environment, individual image captures within an image sequence can receive insufficient light and produce noisy images. The quality of a given image with respect to noise can be improved by utilizing longer exposure durations for each image, but this comes at the expense of a lower frame rate. Alternatively, the image quality with respect to noise can be improved by combining pixels through the technique of binning; however this improvement comes at the expense of lower spatial resolution. In many cases the spatial and temporal resolution of the image sequence are limited by the readout capabilities of the sensor. A sensor is capable of reading a certain number of pixels per second. This readout capability is balanced between spatial and temporal resolution of the readouts. Increasing one must come at the expense of the other in order to keep the total number of pixels read within the achievable range of the sensor.
Many solutions have been proposed to allow a digital image sequence capture device to capture image sequences with improved quality and resolution. One method to reduce noise in a digital image sequence is through temporal noise cleaning. An example of such a technique is given in U.S. Pat. No. 7,330,218. Temporal noise reduction techniques exploit the high temporal correlation among neighboring images to achieve noise reduction. In static scenes, multiple readouts of the same image scene content are available in successive images, allowing for effective noise reduction. The drawbacks of temporal noise reduction include memory requirements to buffer multiple images, as well as computational requirements to filter the images, in particular if motion estimation and compensation are used to align regions of local or global motion. Additionally, temporal noise reduction does nothing to improve the spatial or temporal resolution of the image sequence.
One method to improve temporal resolution is temporal frame interpolation. Those skilled in the art will recognize, however, that such techniques are computationally complex, memory intensive, and often generate artifacts in the interpolated frames.
One method to improve the spatial resolution of an image sequence is through super-resolution techniques. Examples of super-resolution algorithms are provided in U.S. Pat. Nos. 7,215,831 and 7,379,612. Video super-resolution techniques use neighboring frames to estimate each high-resolution video frame. The drawbacks of spatial video super-resolution include computational complexity and memory requirements. Dynamic scenes are also difficult for spatial super-resolution algorithms to process.
Another method to improve the quality of a digital image sequence is through the use of a dual-sensor camera. Such a system is proposed in US Patent Application 2008/0211941, “Digital Camera Using Multiple Image Sensors to Provide Improved Temporal Sampling.” Improved temporal resolution can be achieved by staggering the exposures of the dual sensors Improved image quality and noise reduction are possible by exposing the two sensors equally and then combining the resultant images. The drawbacks to this solution include the costs associated with a dual sensor camera. Additionally, in a dual-lens device, the need exists to spatially align images captured from the different lens systems.
Another method to improve spatial resolution is by capturing an intermittent high resolution image along with the low resolution image sequence, followed by processing to generate an entire high resolution image sequence from the aggregate data. Examples of such solutions are U.S. Pat. Nos. 7,110,025 and 7,372,504. The drawbacks of this solution include in some cases the requirement of an additional sensor and other hardware to capture the high resolution image without disrupting the image sequence capture process. Other drawbacks include the need to buffer multiple images, depending on the frequency and usage of the high resolution images in generating the final high resolution image sequence.
Another method for improving the quality of an image sequence is through the use of an image sensor with improved light sensitivity. Many image sensors use a combination of red, green and blue color filters arranged in the familiar Bayer pattern, as described in U.S. Pat. No. 3,971,065. As solutions for improving image capture under varying light conditions and for improving overall sensitivity of the imaging sensor, modifications to the familiar Bayer pattern have been disclosed. For example, commonly assigned U.S. Patent Applications Publication No. 2007/0046807 entitled “Capturing Images Under Varying Lighting Conditions” by Hamilton et al. and Publication No. 2007/0024931 entitled “Image Sensor with Improved Light Sensitivity” by Compton et al. both describe alternative sensor arrangements that combine color filters with panchromatic filter elements, interleaved in some manner. With this type of solution, some portion of the image sensor detects color; the other panchromatic portion is optimized to detect light spanning the visible band for improved dynamic range and sensitivity. These solutions thus provide a pattern of pixels, some pixels with color filters (providing a narrow-band spectral response) and some without (unfiltered pixels or pixels filtered to provide a broad-band spectral response).
Using a combination of both narrow- and wide-spectral band pixel responses, image sensors can be used at lower light levels or provide shorter exposure durations. See Sato et al in U.S. Pat. No. 4,390,895, Yamagami et al in U.S. Pat. No. 5,323,233, and Gindele et al in U.S. Pat. No. 6,476,865. Such sensors can provide improved image quality at low light levels, but additional techniques are required to address the need for producing image sequences with improved spatial and temporal resolution.
In digital imaging, it is also desirable to capture an image sequence having high dynamic range. In photography and imaging, the dynamic range represents the ratio of two luminance values, with the luminance expressed in candelas per square meter. The range of luminance human vision can handle is quite large. While the luminance of starlight is around 0.001 cd/m2, that of a sunlit scene is around 100,000 cd/m2, which is one hundred million times higher. The luminance of the sun itself is approximately 1,000,000,000 cd/m2. The human eye can accommodate a dynamic range of approximately 10,000:1 in a single view. The dynamic range for a camera is defined as the ratio of the intensity that just saturates the camera to the intensity that just lifts the camera response one standard deviation above camera noise. In most commercially available sensors today, the maximum ratio of signal to noise for a pixel is about 100:1. This, in turn, represents the maximum dynamic range of the pixel.
Since most digital cameras are only able to capture a limited dynamic range (the exposure setting determines which part of the total dynamic range will be captured), high dynamic range images are commonly created from captures of the same scene taken under different exposure levels. For most daylight outdoor scenes excluding the sun, three exposures spaced by two exposure values apart are often sufficient to properly cover the dynamic range. However, this method requires a scene that does not change between the captures in the series.
Jones (U.S. Pat. No. 6,924,841 B2) discloses a method for extending the dynamic range of a sensor by having two groups of pixels with different sensitivities. However, Jones requires that the sensitivity of the first group of pixels overlaps with the sensitivity of the second group of pixels in order to have some common dynamic range. This method is not desirable because it will not provide a substantial dynamic range for real world scenes. It also requires a specialized sensor with pixels of different sensitivities.
Kindt et al. in U.S. Pat. No. 6,348,681 discloses a method and circuit for setting breakpoints for a sensor to achieve a user selected piecewise linear transfer function.
Ando et al. in U.S. Pat. No. 7,446,812 discloses a method for using dual integration periods during a same frame and readout to increase the dynamic range for a capture. This method does not utilize every photon that reaches the sensor because the pixels with shorter integration time will not capture photons between the time of the readout of those pixels and the pixels with the longer integration time.
Thus, there exists a need for producing a digital image sequence with improved image quality, spatial resolution and temporal resolution, without generating spatial or temporal artifacts, and without significant memory costs, computational costs, or hardware costs.
There also exists a need for producing a high dynamic range image from an image sensor without fundamentally increasing the complexity or composition of the individual pixels in the sensor.